Measuring National Immunization System Performance: A Systematic Assessment of Available Resources

This review of monitoring and evaluation resources identified multiple performance indicators that can inform country-specific approaches to evaluating immunization systems.


INTRODUCTION
S uccessful immunization programs are strongly associated with lower childhood morbidity and mortality and are an important marker of health systems performance. [1][2][3] As an essential health service, immunization performance is routinely measured by vaccination coverage. 1 Despite the increased breadth of protection through new vaccine introduction, coverage rates of vaccines given in infancy have stagnated in the last decade and remain lower than the World Health Organization's goal of 90%. 4,5 Immunization successes gained globally between 2010 and 2019 were adversely impacted by the COVID-19 pandemic. 6,7 In 2020, coverage rates dropped to 83% for the third dose of diphtheria-tetanus-pertussis vaccine (DTP3) and 84% for the first dose of measles-containing vaccines 8 and then even further in 2021 (DTP3: 81%; first dose of measles-containing vaccine: 81%). 9 The number of zero-dose children, defined as infants who did not receive any doses of the DTP vaccine, was reported as high as 18.2 million in 2021. 10 Measuring vaccine and zero-dose coverage alone is insufficient to drive improvements in immunization. An immunization system contains health system components, including all the organizations, institutions, resources, processes, and activities involved in the delivery of vaccines. 11 Vaccines of assured quality must be available at the point of service, trained health workers must administer vaccines, and the community must demand vaccines and be able to access them. 1 Program delivery is underpinned by dedicated financing and governance structures and surveillance mechanisms. 1 High-quality and timely data inform where systematic weaknesses exist and how resources should be invested. 1,12 System components need to work together effectively to achieve high vaccine coverage. 1,13 In rural Madagascar, for example, interventions to improve infrastructure, staff training and availability, and procurement systems, underpinned by data from a new health information system platform, increased full vaccination coverage among children aged 12-23 months from 34.6% in 2014 to 63.6% in 2018. 14,15 Initiatives like these demonstrate the importance of assessing immunization system components to target and prioritize strategies to improve overall performance.
Unlike immunization coverage, there is no clear consensus on how to measure the performance of immunization system components. Several monitoring and evaluation (M&E) resources exist to measure the performance of immunization systems or programs and are used for different purposes. [16][17][18] In planning for the integration of COVID-19 vaccination programs into immunization systems, there is a strong focus on integrating not just health services but also health governance functions and on improving the availability and reach of health services across the life course. 19 There are opportunities to capitalize on the investments made during the COVID-19 pandemic, such as through leveraging technological advancements, enhancing disease surveillance and monitoring of adverse events following immunization, enumerating and upskilling workforce, and strengthening supply chains for vaccines and other medications. 19 Examining the current performance of immunization systems can help countries understand how best to achieve integration and leverage the investments made for the COVID-19 response. This creates a timely opportunity for the review of existing M&E resources. In this study, we examine how immunization system performance has been measured in the past and where gaps exist in measuring the performance of specific components of the system.

METHODS
We reviewed existing M&E resources used for immunization systems globally to understand how immunization system performance is measured. We were guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. 20

Conceptualization of an Immunization System
We applied the World Health Organization's (WHO) health system framework to categorize indicators by the components of immunization systems that they intend to measure. The framework describes health systems according to 6 building blocks 21 and is frequently used to examine system-wide impacts of initiatives like the introduction of a new vaccine. 22,23 The 6 building blocks include workforce, information systems, supply chains, financing, governance, and service provision. Additionally, we included safety and regulation of vaccines and demand generation in our framework because the published literature recognizes these components as critical. 24,25 We defined the outcomes of immunization systems as the immediate and short-term effects achieved through the collective and synchronized functioning of immunization system components, namely, to vaccinate populations against a broad range of vaccine-preventable diseases. Impacts of immunization systems were defined as health effects and changes in disease burden associated with vaccine-preventable diseases and overall effects on the health of populations. This conceptualization was the basis for the immunization system framework developed as part of this study (Supplement 1).

Search Strategy
We searched for M&E resources on immunization systems in the peer-reviewed (OVID MEDLINE) Examining the current performance of immunization systems can help countries understand how to achieve integration and leverage COVID-19 response investments.
Measuring National Immunization System Performance www.ghspjournal.org

M&E Resources of Immunization Systems
We included contemporary M&E resources published in 2000 or later that examined systemwide immunization performance. Our selection process is illustrated in the Figure, and the inclusion and exclusion criteria are summarized in the Box. We considered a resource eligible if it (1) incorporated national or country-level indicators that measured the overall performance of the immunization system and at least 1 of its components or (2) examined the performance of multiple components of the system. Resources that included only outcome (e.g., vaccination coverage) or impact-level indicators (e.g., disease incidence or mortality) or that included indicators only at the global, regional, or subnational levels were excluded. We excluded resources that focused on a single aspect of the immunization system (i.e., did not examine overall performance or other components of the system) and studies that evaluated a specific intervention or targeted initiative that did not examine system-wide performance (e.g., implementation of a new logistics management system that only evaluated the direct benefits of supply chain management improvement, without consideration of service provision or coverage of vaccines). Similarly, we excluded M&E resources that were focused on evaluating a disease-specific program and did not include indicators examining the impact of the program on the immunization system. Titles and abstracts were screened by 1 author (CP) using Rayyan QCRI. 26 Ambiguous resources were discussed with MS. Two authors (CM and RF) who are content area experts also reviewed the list of resources identified to ensure it was comprehensive.

Immunization System Indicators
Following extraction of all indicators from the included M&E resources, we included all indicators where the value was numeric or categorical (e.g., yes/no or high/medium/low responses). We included indicators measuring performance at a global level that could be modified to the national level, as data for these indicators are typically

FIGURE. Search Results and Included Resources on Immunization Monitoring and Evaluation
Measuring National Immunization System Performance www.ghspjournal.org collected nationally and then aggregated to obtain a global estimate (e.g., "proportion of countries with DTP vaccine coverage ≥80%" was extracted as "national DTP vaccine coverage is ≥80%"). We excluded indicators that were not applicable at the national level related to goals requiring global effort and coordination, such as development of new vaccine platforms. In our examination of distinct indicators, we removed indicators that were duplicated across resources.

Data Extraction and Analysis
We tabulated data on the M&E resources, including the tool's author, year of publication, purpose, context or location where it was used, and the number of indicators in each tool. We used methods previously used to summarize and compare performance indicators, particularly those used to analyze indicators for health system performance. 27,28 We extracted and classified indicators into 10 broad domains aligning with system impacts, system outcomes, and the 8 system components. We further categorized the indicators within each domain into measurement areas to better understand how performance within each domain is assessed (see Supplement 1 for definitions). A formal quality assessment of resources and indicators was not done. Data were extracted using Microsoft Excel and imported into NVivo 12 for coding. Two authors (CP and NR) independently coded the indicators. Discrepancies were resolved through discussion and consensus. We categorized each indicator into a single domain and measurement area.
Where indicators could be categorized under 2 or more system domains, we used contextual information from the source document to understand which domain the indicator was intended to measure. For example, "the proportion of the immunization budget dedicated to advocacy activities" was categorized under demand generation rather than financing because it was used in the source document to measure commitment to engaging with communities to build demand for vaccines.
We used summary statistics to calculate (1) the number of M&E resources, including indicators for each domain and measurement area, and (2) the number of indicators in each domain and measurement area. We summarize our findings narratively.

RESULTS
After screening 3,707 titles in the peer-reviewed literature and 1,000 in the gray literature, we identified 20 M&E resources that met the inclusion criteria (

Inclusion criteria
Examines system-wide performance of immunization system or the system-wide impact of a disease-specific immunization program or initiative Includes performance indicators on overall system performance and at least 1 of the system components or examines the performance of multiple components of the system Incorporates national-level indicators

Exclusion criteria
Included only outcome (e.g., vaccination coverage) or impact-level indicators (e.g., disease incidence) Did not include national-level indicators (e.g., resources measuring performance at the subnational level only) Focused on a single component of the immunization system without examining system-wide performance Evaluated a disease-specific immunization program, intervention, or targeted initiative without examining its impact on system-wide performance Measuring National Immunization System Performance www.ghspjournal.org  Indicators that can be used to monitor immunization performance Peer-reviewed literature; provides examples of multiple indicators necessary to monitor various components of immunization programs and to assess overall program performance.
All countries, but particularly LMICs 21 WHO, 2020 13 Immunization Agenda 2030 Monitoring and Evaluation Framework Gray literature; to measure progress toward the goals and objectives of IA2030 strategy and enable use of data for action to continuously improve immunization programs at the national, regional, and global levels.

All countries 112
Gavi, 2018 17 Gavi 2016- Gray literature; to assess ability of the REC-QI approach to improve functionality, efficiency, and sustainability of routine immunization system and to assess how key components of REC-QI model contribute to strong routine immunization system.
LMICs (with a focus on African countries) 19 WHO, 2010 33 New Vaccine Post-Introduction Evaluation (PIE) Tool Gray literature; to evaluate overall impact of introduction of a new vaccine on country's national immunization program.

All countries 16
Griffith et al., 2010 34 Toolkit for assessing the impact of measles eradication activities on immunization services and health systems Gray literature; to assess impact of measles elimination activities on goals related to elimination of measles and impact on routine immunization services and the health system.
Global with a focus on LMICs, especially countries with measles elimination programs 11 WHO, 2002 35 WHO Common Assessment Tool for Immunization Services Gray literature; to assess immunization services in the wider context of the health system.

All countries 55
Gavi, 2002 36 Gavi and WHO Monitoring National Immunization Systems Using Core Indicators Gray literature; to monitor progress toward immunization system targets, and to identify and analyze problems that can guide management decisions. resource, context where the resource is intended to be used, and the number of indicators. All resources were designed to monitor progress toward immunization program objectives and collate data to guide national-level strategic and programmatic decision-making to achieve goals.
Three resources assessed the extent to which specific initiatives (namely, measles and rubella elimination programs and polio eradication programs) improved routine immunization. 34,37,42 Three resources examined the extent to which the objectives of global development organizations were Gavi indicators for sustainable immunization systems Peer-reviewed literature; to analyze the sustainability of immunization programs in Gavi transitioning countries and identify potential sustainability issues, particularly in 4 programmatic areas: (1) decision-making, (2) political commitment and financial sustainability, Indicators for Immunization Systems Management Group Routine Immunization Dashboard Peer-reviewed literature; to monitor progress in routine immunization through a dashboard using agreed standard indicators that reflect steps in the pathway of routine immunization strengthening with a focus on polio high-risk districts.
LMICs, especially those with polio eradication programs 15 Tegegne et al., 2016 42 Accountability framework for the Nigeria polio program Peer-reviewed literature; to systematically monitor and evaluate the impact of the polio eradication initiative in Nigeria using indicators that cut across different program areas and to measure program and staff performance.

Nigeria 21
Shuaib et al., 2014 43 Accountability Framework for Routine Immunization, Nigeria Peer-reviewed literature; to monitor routine immunization administration and vaccine management and to ensure that government officials could access highquality routine immunization data to monitor performance and improve routine immunization coverage.

Nigeria 21
WHO, 2013 44 Global Vaccine Action Plan: Monitoring and Evaluation/ Accountability Framework Peer-reviewed literature; to monitor progress of immunization programs against Global Vaccine Action Plan goals, specifically monitoring results.

All countries 27
Abbreviations: EPI, Expanded Programme on Immunization; IA2030, Immunization Agenda 2030; JRF, Joint Reporting Form; LMIC, low-and middle-income country; MCSP, Maternal and Child Survival Program; SIA, supplementary immunization activity; WHO, World Health Organization. a Where information was available, the year of publication denotes the year that the indicator tool was published and available for use and is not necessarily the date of publication of the article/report. b The JRF is revised on a regular basis; this study includes indicators included in the tool in 2019.
achieved. 13,17,44 One resource (WHO's New Vaccine Post-Introduction Evaluation Tool) evaluated the introduction of new vaccines into national immunization programs. 33 One resource established for use in the United States 40 included indicators that could be applied in other countries but is likely to be of limited use in low-and-middle-income countries (LMICs) due to the differing priorities and degree of maturity of immunization systems compared with highincome countries. The 20 included resources captured a total of 740 indicators, of which 631 distinct indicators (85%) were retained following exclusion of ineligible and duplicate indicators (Figure and Box). Table 2 presents the number of resources and corresponding distinct indicators covering each domain. Of the 631 distinct indicators, 47 (7.4%) measured system impacts, 124 (19.7%) measured system outcomes, and 460 (72.9%) measured the performance of immunization system components. Three resources-the Immunization Agenda 2030 (IA2030) Monitoring and Evaluation Framework, 13 the WHO-UNICEF Joint Reporting Form, 29 and the guide for conducting an Expanded Programme on Immunization review 31 -included indicators that covered all 10 domains. Indicators from these 3 resources comprised almost half of all 740 indicators identified (112, 169, and 85, respectively, a total of 366/740 [49.5%]). A summary of which resources included indicators that measured performance in each domain is in Supplement 3. Table 3 summarizes the resources and indicators measuring the outcomes (n=8) and impacts of immunization systems (n=16). Nine resources (45.0%) included indicators measuring system impact, with the majority (39/47 [83.0%]) of indicators measuring incidence, mortality, or disability-adjusted life years attributable to specific vaccine-preventable diseases. In our review, indicators for immunization system outcomes, found in 17 (85.0%) resources, comprised 19.

Performance Indicators for Immunization System Components
The number of resources and indicators measuring the performance of immunization system components is summarized in      Indicators for processes for monitoring and evaluation examined if measures for ensuring accountability in immunization programs were in place (e.g., through an evaluation framework or other means to monitor performance, provision of feedback to subnational levels, and engagement of various stakeholders in monitoring immunization programs).

Information systems
Any 18 (90.0) 92 (100.0) Indicators for data quality included those measuring completeness and timeliness of reporting to higher levels of the health system, the accuracy of data (e.g., through coverage rates greater than 100% or negative values for coverage dropout rates or if denominator data are accurate) and assessing if data on certain variables is collected (e.g., age data on cases of VPDs). Indicators for disease surveillance examined the ability to detect specific diseases of interest such as polio, measles, and rubella, including laboratory capacity and capability to test for VPDs. Indicators on processes and systems examined whether certain systems existed to collect and report data on immunizations and how widespread access to them was (e.g., the proportion of the population with access to immunization records). This included 7 indicators on the use of technology such as digital tools (e.g., electronic stock management system) or electronic means of entering and transmitting data. Indicators on data use examined whether data was used to inform plans for delivering routine immunization services and in outbreak response campaigns. Indicators for safety surveillance examined whether systems to detect adverse events related to immunization existed and the number and rate of adverse event reports. Indicators assessing regulatory policies and processes largely examined whether certain policies and procedures to ensure the safety of vaccine products (e.g., the proportion of vaccines procured of assured quality or existence of guidance on waste management) and vaccine administration (e.g., completion of a standardized injection safety assessment) were in place.  Indicators examining effective management of vaccines related to cold chain management (e.g., proportion of facilities with functional refrigerators or with temperature monitored), supply chain management (e.g., use of vaccine forecasting processes and kilometers per vehicle to transport vaccines) and management of waste (e.g., wastage rates of closed vials and availability of adequate infrastructure and supplies for waste management). Indicators related to innovations in vaccine products examined the use of innovative products (e.g., newly recommended vaccines or new technologies to deliver vaccines) and capacity to conduct vaccine research, particularly clinical trials. Indicators on availability of health workers assessed the size of the workforce, focusing on the clinical workforce (e.g., number of health workers per 10,000 population) and gaps in workforce (e.g., the ratio of unfilled to total number of posts). Three indicators examined whether there were adequate workforce for nonclinical immunization functions, (i.e., data management, human resources, and supply chain management). Indicators on health worker competency examined the proportion of staff who were able to conduct certain tasks correctly (e.g., providing correct case definition for a disease).  Those on health worker availability largely measured the density of the clinical workforce; 3 assessed the availability of workforce supporting nonclinical functions in the immunization system (specifically data management, human resources, and supply chain management), but none assessed the size or capacity of the public health workforce.
Indicators measuring the performance of regulation and pharmacovigilance and demand generation were the least frequent (

DISCUSSION
Our study found a multitude of indicators examining the performance of immunization systems and their underlying components, with 631 distinct indicators across 20 M&E resources. We identified variations in how domains were measured, including those where there was some consistency across resources, like those for coverage, service provision, and vaccine supplies and logistics. Although some differences may be due to the differing purposes of the resources or operational contexts in which they are used, the differences and multiplicity of resources reflect the increasing complexity of immunization programs and the changing policies, priorities, and reporting requirements of immunization actors at the global level. Recently, global focus has shifted from achieving high coverage of individual vaccines through disease-specific initiatives to achieving universal health coverage through system-wide strengthening approaches. 5,45 Zero-dose coverage has notably emerged as a metric for childhood immunization performance, featured prominently in the IA2030 framework and Gavi's 2021-2025 strategy. 5,13,46 Global attention is also increasing toward expanding immunization across the life span, but our review found that indicators measuring vaccination coverage in adolescence and adulthood are limited in number and scope. Fourteen of 17 indicators examining vaccination after early childhood were identified from the WHO-UNICEF Joint Reporting Form, and 11 of these pertained to influenza vaccine uptake in high-risk populations. COVID-19 vaccination is the first largescale program targeting adults globally. Before this, 62% (120/194) of countries reported having at least 1 adult immunization program, but highand upper-middle-income countries were almost 22 times more likely to have such a program compared with low-and lower-middle income countries. 47 Our review did not identify any metrics examining the burden of well-established diseases affecting older adults, like influenza, pneumococcal disease, and herpes zoster. Indicators on both disease incidence and disease surveillance systems are focused on childhood diseases and diseasespecific programs, such as polio eradication and measles elimination. Lack of data on the disease burden and the potential public health impact of vaccinating against these diseases in LMICs hinders decision-making related to introducing these programs. 47,48 Enhancing surveillance for diseases affecting adults and reporting these data stratified by age and sex will require substantial investment, political will, and advocacy at global and national levels, but it is a necessary precursor to introducing adult vaccination programs in LMICs.
We found that the processes by which immunization data are collected, analyzed, reported, and accessed, and the systems used to do so, are not consistently measured across resources. Most indicators examined whether a system was in place but not its functionality, acceptability, or The differences and multiplicity of identified resources reflect the increasing complexity of immunization programs and global changes in policies, priorities, and reporting requirements.
Measuring National Immunization System Performance www.ghspjournal.org Global Health: Science and Practice 2023 | Volume 11 | Number 3 use. Experience with interpreting data collected and reported at a global level highlights the data quality issues that arise. For example, data on 14 of the 27 indicators proposed to measure progress against the goals of the Global Vaccine Action Plan (which preceded the IA2030 strategy) were difficult to interpret due to poor data quality and variability in reporting across countries. 49 One indicator in the plan, "immunization coverage data assessed as high quality by WHO and UNICEF," was abandoned due to an inability to find a suitable measure. 49 Data on denominators based on births and deaths statistics are known to be inaccurate and incomplete in LMICs, particularly at subnational levels. 12,50 In 2016, 76 of 96 countries reported at least 1 district with DTP3 vaccine coverage greater than 100%. 51 Unreliable data quality means that wide variations are difficult to interpret, as it is unclear if the variations are an artifact of the data or if they are true variations. 49 Noting the importance of robust information systems to track coverage and other immunization targets, 12,50 newer resources are available such as the Data Quality Self-Assessment tool developed by the WHO Immunization Analysis and Insights Unit. 52 Despite improved data quality over the last 2 decades, gains were not universal, with resource-constrained countries and those with lower immunization performance continuing to have limited to poorer quality data. 53 Although data on numerous indicators are often collected and reported by countries, we identified few indicators in the resources included in our review that measured data-driven decisionmaking and program planning at the national level. Availability of data does not necessarily translate into action; mechanisms and accountability frameworks to incorporate data into decision-making are needed. Interventions to improve data use can drive improvements in data quality and increase demand for better data. 54 A systematic review of the DHIS2 in 11 countries found that access to data increased a sense of ownership and responsibility for the quality of data, fostering a culture of data use and improvement. 55 Although the need for including indicators on data use in M&E resources is clear, identifying user-friendly, valid indicators for data use has proven difficult, with no standard or widely accepted approach to defining or measuring data use available. 56 Achieving consensus on what constitutes data use can help to develop performance metrics to measure data use and is an area for future research, particularly given the recent focus on improving health information systems through the implementation of digital technologies. One tool that addresses this, albeit within a specialized system component, is the Effective Vaccine Management assessment tool (excluded from our review for reasons described below), as its indicators provide the basis for recommending actions that can lead to cold chain or other supply chain improvements. Although our review focuses on the national level, data use at subnational and local health facility levels is critical to improving coverage, for example, through tracking and identifying defaulters and under-vaccinated populations, improving vaccine supply management and planning of immunization services, and tailoring strategies to build demand for vaccination. 12 With the increasing focus on improving the availability and use of subnational-level data by organizations like Gavi, 57 the need for performance metrics measuring data use will likely grow.
We identified few indicators measuring the performance of regulation and pharmacovigilance systems for vaccines, likely because efforts to implement and strengthen national safety surveillance for vaccines are relatively more recent compared to initiatives targeting other components of the immunization system. The impetus to introduce and enhance safety surveillance systems increased for COVID-19 vaccination. 58 The rate of adverse events following immunization (AEFI) reporting, measured as the annual AEFI reports per 100,000 surviving infants, has been proposed as a suitable indicator to monitor the performance of AEFI surveillance systems 59 and was 1 of the more common of the 9 indicators for safety surveillance identified in this study. The number of countries meeting this reporting target has grown from 80 (41% of 194 countries) in 2000 to 109 (56%) in 2019 but varied by region. 60 However, more granular indicators that more accurately capture the quality, functionality (e.g., ability to assess causality of AEFIs), and effectiveness of these systems are needed. We also did not identify any indicators examining the existence or functionality of no-fault vaccine injury compensation schemes, although the call to implement these, particularly in countries with more advanced safety surveillance systems, is growing. 61 Our review found that few M&E resources measuring the performance of immunization systems include indicators that measure the confidence in vaccines or social and behavioral drivers of demand for immunization, despite growing concerns over vaccine hesitancy. 62 Data on how countries routinely and systematically track vaccine confidence is currently lacking. Measuring vaccine demand has proved to be a challenge; the Although countries collect and report data on numerous indicators, we identified few indicators for data-driven decision-making and program planning at the national level.
Measuring National Immunization System Performance www.ghspjournal.org indicators identified in this review are limited in their interpretation and utility, as it is unclear how concepts such as "supports vaccination" or "lack of confidence" were defined. The lack of consistency in definitions is echoed in a review examining how outcomes in trials of childhood vaccination communication interventions are measured. 63 Composite indicators based on qualitative data collected via the WHO-UNICEF Joint Reporting Form were the primary way to monitor vaccine demand globally 64 but were criticized for the lack of clarity in defining what was being measured. 49 In another review, 12 of 14 measures of confidence in childhood vaccination identified were developed and validated in high-income countries. 65 The recently published resources for behavioral and social drivers of vaccination 66 provide a framework to assess reasons for undervaccination, which may address the gaps and lack of consistency in metrics for this domain and can contribute to formative work on program strengthening. Indicators examining equitable coverage largely focused on disparities by geographical areas (such as districts), and 5 examined coverage by socioeconomic status. Our review identified some disease burden indicators disaggregated by sex but none for coverage or any that examined or collected data on the role of gender and diversity in making decisions about or implementing immunization programs. Despite known differences in immune responses to certain infections and adverse events following immunization, studies of vaccine effectiveness and safety often do not report results by gender. 67,68 Evidence suggests that gender influences health status, access to resources and health services, decision-making autonomy, and the quality of health services. 69 Yet, only 5 of 58 countries with COVID-19 vaccine policies in March 2021 referred to gender, and 34 of 180 countries reported sex-disaggregated data on COVID-19 vaccine coverage between April and May 2021. 70 The absence of indicators in our study on disability, diversity, and inclusion was also a gap. Achievement of the IA2030 goals of reducing zero-dose children and improving equity of coverage across the life span will require identifying predictors of low coverage and selecting pro-equity strategies to address disparities, particularly in the context of COVID-19related disruptions to immunization that have disproportionately affected poorer and more vulnerable populations. 71 The behavioral and social drivers of vaccination tools 66 help to fill this gap, as they collect data on gender and other demographic data to help explain what drives vaccine uptake.
Further work to address this gap is needed, as is the need to understand the role of gender, diversity, and inclusion in strengthening decisionmaking about immunization.
Although 11 of the 17 indicators examining integration of immunization services related to co-delivery of immunization alongside other health services (e.g., antenatal care) or in nontraditional settings (e.g., schools), it is unclear if this is the best way to measure integration of primary care services. 72 True integration to achieve universal health coverage requires integration across the system in planning, financial resourcing, training and supervision, and community engagement, and current indicators to measure integration do not reflect this. 73 The use of an integration index has been proposed as a way forward 72 but runs the risk of measuring concurrent performance across programs without actually measuring the extent to which programs are integrated. Future efforts to define indicators should also consider quantifying the health and efficiency gains that are expected to occur following integration.
The M&E resources in this review did not identify any indicators examining the capacity or competency of the public health workforce or the surge clinical capacity available to draw on in the event of a public health emergency. The need for defining, mapping, and measuring the workforce, including the public health workforce, is recommended in WHO's Global Strategy on Human Resources for Health 2030. 74 A framework of immunization workforce competencies, such as WHO's roadmap for public health workforce developed by Traicoff et al., 25 can help countries to establish what the minimum skills and competencies needed are and track suitable indicators and numbers of staff trained in those technical competencies. The Joint External Evaluation Tool, which aims to support countries to develop their capacity to prevent public health threats, provides additional metrics and targets for the public health workforce and surge capacity required, particularly in the context of a public health emergency. 75 Incorporating these core competencies into public health and clinical training programs can secure surge capacity in the event of future acute emergencies and increase the skills mix of primary health care providers, contributing toward the goal of universal health coverage.
Our study identified 47 distinct indicators across 9/20 (45.0%) M&E resources that measure the impact of immunization programs on disease burden and elimination targets. This small number is likely due to (1) our inclusion criteria and The absence of indicators in our study on disability, diversity, and inclusion was a gap that requires further work to address.
Measuring National Immunization System Performance www.ghspjournal.org focus on metrics assessing immunization system performance and (2) exclusion of resources that did not measure the operational aspects of immunization systems. Although the indicators identified disease-specific and some summary metrics of disease burden and disease elimination and eradication targets, metrics evaluating wider economic and societal benefits, such as reductions in poverty, improvements in productivity, and financial returns on investment, 76 were absent. Including these metrics in evaluations of immunization system performance can help to build the case for financial investment in immunization systems and assist decision-making through comparisons with other initiatives or packages of interventions. However, the societal and financial impacts of immunization have often been estimated through modeling studies, 77,78 and it may be challenging for resource-restricted countries to conduct these analyses in the short term.

Strengths and Limitations
To our knowledge, this is the first published review of indicators used to measure system-wide performance of national immunization systems that includes M&E resources published over 2 decades. This review intentionally focuses on M&E resources used globally. Previous global immunization strategies were viewed as top-down strategies, but there is growing recognition that countryspecific goals and targets are needed. 4 Our review collates the indicators used to measure immunization system performance and provides a starting point for national governments and partner agencies to consider how best to assess performance in their own contexts. Additionally, it draws attention to gaps in indicators used to date, highlighting where development partners and researchers can focus their efforts to develop and validate indicators. Importantly, our study highlights the large burden of data collection and routine reporting for immunization alone, with 169 indicators identified in the 2019 WHO-UNICEF Joint Reporting Form and 112 indicators for national monitoring in the IA2030 M&E framework. We removed 93 duplicate indicators identified in our review but were still left with 631 indicators. Many indicators measured the same construct but in slightly different ways, which means that data collected with 1 resource may not be appropriate for use with another resource, resulting in duplication of efforts if reporting on those indicators is required by development or financial partners. The workload associated with collecting and reporting data using different M&E resources is significant, especially in resourceconstrained settings, where dedicating resources to meet reporting requirements has opportunity costs. 12,79,80 Data collection, analysis, and reporting require dedicated financing, resourcing, and accountability mechanisms, potentially diverting resources from other areas of the health system. Countries with constrained systems likely have a greater proportion of their immunization system costs covered by donors and international partners and thus have greater data reporting responsibilities despite having fewer resources to do this. Previous M&E resources have also taken a top-down approach that does not account for variability in country contexts, resulting in reduced country commitment to achieving global immunization goals. 4 Further research and coordination on which indicators are most important to measure uniformly across all countries is needed, with clear justification of how these indicators are linked to public health benefits to justify resourcing to collect and report this data. For the remainder, it may be more suitable to identify and focus on key indicators based on the values and challenges of the local context, which aligns with the IA2030 strategy's call for countries to select indicators based on their needs. 13 Although we systematically searched the literature, we identified only 6 peer-reviewed publications. This is unsurprising as the majority of widely used M&E resources are published in the gray literature by global partner agencies. The Medical Subject Headings used in the MEDLINE database for this topic area are not well-defined and provide a substantial volume of false results. A further limitation is that we only included resources that were publicly available, whereas many countries may report on performance directly to their funders. We ensured that our search strategy captured the Medical Subject Headings against which the identified articles were indexed, and 2 authors of this article are content area experts who checked that we included all major or influential M&E resources. We excluded some widely used and well-established resources, particularly those published by the WHO's Immunization Analysis and Insights group, such as the Effective Vaccine Management assessment tool and vaccine-specific post-campaign assessment resources (e.g., for COVID-19 and influenza vaccines). Some of these were focused on a specific aspect of immunization systems rather than system-wide performance or had a disease-specific or context-specific focus. M&E resources on child health programs, such as Our review provides a starting point for national governments and partner agencies to assess immunization system performance in their own contexts.
Measuring National Immunization System Performance www.ghspjournal.org

MEASURE Evaluation's A Guide for Monitoring and
Evaluating Child Health Programs, 81 were not included due to their focus on child health more broadly, of which immunization is a smaller component, and on measuring outcomes of the system rather than the performance of its components. Nevertheless, these are important assessment resources whose use can contribute toward strengthening immunization system components. We did not conduct a quality assessment of indicators using a standardized quality assessment framework. Not all of the indicators were clearly defined, and few were accompanied by a data dictionary clarifying how to calculate statistics and where to source data. It is possible that some indicators may be interpreted differently depending on the context. Although most resources were developed through iterative and consultative processes, it is unclear if indicators have been validated and are associated with improvements in performance. Thus, the existence of M&E resources and indicators does not necessarily mean they are used or provide insightful information. It is likely that countries are tailoring assessments to suit their contexts and measuring indicators that best match their goals and available resources. Furthermore, performance indicators often have limited use in understanding the drivers of performance. Several M&E resources provide guidance on using other data collection methods, such as interviews and focus groups, to generate information to supplement quantitative indicators that together can inform future actions. Further research is needed to better understand how these evaluation resources are used, whether they drive improvements in performance and the pathways for doing so, and which system-level indicators most closely correlate with improvements in vaccine coverage, equity, and reductions in disease. Measuring implementation strength (i.e., the amount of input or activity to support program implementation), 82 a construct that did not appear in the resources included in our review, may be another way to cumulatively interpret data from selected indicators but requires research to determine which group of indicators can do so most effectively. Developers of M&E resources should assess the quality of indicators included to ensure they are measurable, easy to interpret by users, linked to performance improvements, and relevant to decision-making.
We elected to use the WHO health system framework as the basis for the components of immunization systems, which we used to categorize indicators. This framework has rightly been criticized for focusing on the infrastructural and resourcerelated components of health systems and insufficiently accounting for the interdependencies between components as well as intangible components such as trust and social value. 83 In our analysis of indicators, we found that several indicators could have been classified under more than 1 system component, reflecting the interdependencies and connectedness of the components underlying the immunization system. The indicators we identified did not explicitly examine the interactions between components. As discussed earlier, the quality of the indicators was not assessed, and it is unknown if performance of 1 component is linked to or predictive of performance of another. Nevertheless, this framework provides a useful basis to describe the components that describe the health system and its basic functions and has been used to examine the impact of various initiatives on the immunization and health system. 22,23 Finally, the inductive method we used to identify measurement areas based on the themes that emerged from our review introduced unavoidable risk of measurement and selection bias. As content area experts, we acknowledge that our prior experiences, assumptions, and beliefs have the potential to influence the research process. We have attempted to minimize this bias by having 2 authors independently code indicators and agree on the definitions set out in Supplement 1.

CONCLUSIONS
This review identified a multitude of indicators to measure immunization system performance. We identified heterogeneity in metrics assessing the performance of some immunization system components; further studies are needed to reach consensus on how to measure performance in these areas. This summary of indicators can inform country-specific approaches to measuring system performance moving forward, particularly as countries seek to evaluate the impact of the COVID-19 pandemic and leverage the investments made during the response. As countries look to identify focus areas for improvement, they can benefit from selecting indicators that are aligned with national goals, values, and priorities and establishing accountability frameworks to monitor performance and take action. Establishing a country-focused set of core indicators and improving the quality of data on these select metrics will position decisionmakers to have better access to data that is useful in decision-making, enabling countries to achieve their immunization goals.

Developers of M&E resources
should assess the quality of indicators included to ensure they are measurable, easy to interpret, linked to performance improvements, and relevant to decision-making.